PYTHON-5253 Automated Spec Test Sync #2409

sleepyStick · 2025-06-25T21:51:27Z

Generates at most one weekly PR on Monday's at 9am pst (assuming the evergreen hosts are in utc time...).

At a high level, I've written a script that simply loops through pymongo's existing spec tests and calls the existing bash resync-spec.sh to re-sync all the specs. Then if changes were made, a PR would be created. If all specs are up to date, then no PR will be made.

This PR will be created by mongodb-drivers-pr-bot with the changes. If an error occurred during the syncing of a spec, the PR description will display the name of the spec along with stderr from the bash resync-spec.sh <spec> command.

An example of a generated PR #2407

…thon-driver into PYTHON-5253

sleepyStick

Added some comments that I think might help the reviewer :)

sleepyStick · 2025-06-26T16:44:33Z

.evergreen/resync-specs.sh

-cd $SPECS/source
-make
-cd -
+if ! [ -n "${CI:-}" ]


CI is always defined in an EVG run and we only want to run these things in a local checkout.

sleepyStick · 2025-06-26T16:46:13Z

.evergreen/scripts/create-pr.sh

+#!/usr/bin/env bash
+
+tools="../drivers-evergreen-tools"
+git clone https://github.com/mongodb-labs/drivers-evergreen-tools.git $tools


cloning drivers evergreen tools because there are some scripts (mainly ./secrets-export.sh and get-access-token.sh that are there and i'd rather not copy it over)

Don't we already have a script to clone this repo?

we do? I didn't know that. Do you know what it's called?

.evergreen/scripts/configure-env.sh

Thanks for pointing it out! I just removed the clone here and found the existing path for it!

sleepyStick · 2025-06-26T16:49:24Z

.evergreen/scripts/resync-all-specs.py

+        and len(list(os.scandir(pathlib.Path(entry.path) / "tests"))) > 1
+    }
+    test_set = {entry.name.replace("-", "_") for entry in os.scandir(directory) if entry.is_dir()}
+    known_mappings = {


I think there's a slight difference in how we name the directories vs how the spec names them sometimes? (Not 100% sure if these are correct though, so please let me know if these need to be changed!)

Yes we often have different names than the spec repo, that's expected.

sleepyStick · 2025-06-26T16:58:48Z

.evergreen/scripts/resync-all-specs.py

+        if spec.name in ["asynchronous"]:
+            continue
+        process = subprocess.run(
+            ["bash", "./.evergreen/resync-specs.sh", spec.name],  # noqa: S603, S607


Note:

S603 `subprocess` call: check for execution of untrusted input S607 Starting a process with a partial executable path

S603: I think spec.name is safe because they're only acquired from our directories within test
S607: I think bash is sufficient for executable (they want the whole path)

sleepyStick · 2025-06-26T17:01:43Z

.evergreen/scripts/resync-all-specs.py

+            errored[spec.name] = process.stderr
+
+    process = subprocess.run(
+        ["git diff --name-only | awk -F'/' '{print $2}' | sort | uniq"],  # noqa: S607


Note:

S607 Starting a process with a partial executable path

I tried to separate the string into ["git", "diff --name-only | awk -F'/' '{print $2}' | sort | uniq"] but that caused the command to fail? So I kept it as a whole string..

Each space-separated word has to be a separate arg, keeping it as one here is probably more readable.

sleepyStick · 2025-06-26T17:03:06Z

.evergreen/specs.patch

@@ -0,0 +1,1327 @@
+diff --git a/test/load_balancer/cursors.json b/test/load_balancer/cursors.json


This is a file of known spec test differences that we don't want to be committed. If any one of these is incorrect, or I'm missing some, lmk!

We should add a section to CONTRIBUTING.md detailing the why and how of modifying this file. It should also contain the policy we follow summarized by Shane in the Slack thread.

Good point. Just added a blurb, but lmk if the language is unclear or anything like that. (I feel like my technical writing skills are lacking sometimes)

NoahStapp

Great work! Some minor comments.

NoahStapp · 2025-06-26T18:54:10Z

.evergreen/scripts/resync-all-specs.py

+            ["bash", "./.evergreen/resync-specs.sh", spec.name],  # noqa: S603, S607
+            capture_output=True,
+            text=True,
+            check=False,


What's the motivation behind check=False here and then checking the returncode explicitly instead of using check=True?

err no motivation (I learned about check when ruff yelled at me for it so) Changed it to check=True inside of a try catch!

NoahStapp · 2025-06-26T18:54:47Z

.evergreen/scripts/resync-all-specs.py

+            errored[spec.name] = process.stderr
+
+    process = subprocess.run(
+        ["git diff --name-only | awk -F'/' '{print $2}' | sort | uniq"],  # noqa: S607


Each space-separated word has to be a separate arg, keeping it as one here is probably more readable.

NoahStapp · 2025-06-26T18:56:02Z

.evergreen/scripts/resync-all-specs.py

+        and len(list(os.scandir(pathlib.Path(entry.path) / "tests"))) > 1
+    }
+    test_set = {entry.name.replace("-", "_") for entry in os.scandir(directory) if entry.is_dir()}
+    known_mappings = {


Yes we often have different names than the spec repo, that's expected.

NoahStapp · 2025-06-27T17:58:21Z

.evergreen/specs.patch

@@ -0,0 +1,1327 @@
+diff --git a/test/load_balancer/cursors.json b/test/load_balancer/cursors.json


We should add a section to CONTRIBUTING.md detailing the why and how of modifying this file. It should also contain the policy we follow summarized by Shane in the Slack thread.

blink1073 · 2025-07-11T17:48:33Z

It looks like some tests got unskipped by accident, for example:

https://spruce.mongodb.com/task/mongo_python_driver_mongodb_v4.4_test_server_version_pypy3.10_async_auth_ssl_sharded_cluster_patch_244f17d57b1a28b06756b026372e27bc07b61e67_68713fbc31f3100007df7822_25_07_11_16_45_51/tests?execution=0&sorts=STATUS%3AASC

sleepyStick · 2025-07-11T17:53:44Z

It looks like some tests got unskipped by accident, for example:

spruce.mongodb.com/task/mongo_python_driver_mongodb_v4.4_test_server_version_pypy3.10_async_auth_ssl_sharded_cluster_patch_244f17d57b1a28b06756b026372e27bc07b61e67_68713fbc31f3100007df7822_25_07_11_16_45_51/tests?execution=0&sorts=STATUS%3AASC

Oh I think I just need to merge master into this branch? We'll see if that fixes it

blink1073 · 2025-07-11T20:04:41Z

I opened https://jira.mongodb.org/browse/PYTHON-5439 for the link failure

blink1073

LGTM!

NoahStapp

Great work! Added several suggestions for the CONTRIBUTING.md documentation section to ensure active voice and overall consistency.

NoahStapp · 2025-07-14T15:03:55Z

.evergreen/remove-unimplemented-tests.sh

+#!/bin/bash
+PYMONGO=$(dirname "$(cd "$(dirname "$0")" || exit; pwd)")
+
+rm -rf $PYMONGO/test/server_selection/logging


This should have a comment explaining why we remove this directory.

I took this line from resync-spec.sh (moved it here for consistency) but there was no comment in resync-spec.sh for why its removed..do you happen to know why?

We put those tests in test/server_selection_logging instead.

NoahStapp · 2025-07-14T15:05:13Z

.evergreen/scripts/create-spec-pr.sh

+token=$(bash ./get-access-token.sh $repo $owner)
+if [ -z "${token}" ]; then
+    echo "Failed to get github access token!"
+    popd || exit


If we're going to exit anyway, why popd first?

Good point....turns out i had originally just done a popd and then exit 1, but then my pycharm linter was like "yo, what if popd fails? you should add an || exit to be safe" so i did LOL but its kinda unnecessary cuz I know I did a pushd at the beginning of the script.
anyways, i removed the || exit

wait jk, its not just pycharm linter, our liters say that too...(but it was fine when i ran it locally? 😕)
edit: i realized i didn't actually answer your question...i wanna popd before i exit so that the resync-all-specs.sh is at the path it expects to be on (i delete a file that the script generated to "cleanup" after calling this create-pr script)

CONTRIBUTING.md

Co-authored-by: Noah Stapp <[email protected]>

…thon-driver into PYTHON-5253

NoahStapp

Comment needed for test/server_selection_logging.

sleepyStick · 2025-07-14T17:35:00Z

Comment needed for test/server_selection_logging.

Moved that line back to resync-spec.sh and added a comment!

sleepyStick and others added 19 commits May 7, 2025 10:14

WIP

fe1b07d

trying to get it to work locally?

f822f31

wip

6909518

still a wip

73139a4

WIP

b3a2b29

trying to get it to work locally?

49685ae

wip

076ed93

still a wip

07ea1f6

add patch file, use comment bot, still a wip

f4b3ddc

Merge branch 'PYTHON-5253' of https://github.com/sleepyStick/mongo-py…

547dd76

…thon-driver into PYTHON-5253

Merge branch 'mongodb:master' into PYTHON-5253

a97d5f4

fun fact, using a python script is actually *way* easier

925aeb5

remove comment

67b995a

remove comment pt2

53e8945

9am pst is 4pm utc i think?

9f6f727

remove unused vars and cleanup debugging prints

3144383

oops accidentally deleted this line

2bea24b

fix resync commit msg

4bef7ca

add / fix comments

8aaf791

sleepyStick commented Jun 26, 2025

View reviewed changes

sleepyStick marked this pull request as ready for review June 26, 2025 17:04

NoahStapp requested changes Jun 26, 2025

View reviewed changes

sleepyStick and others added 2 commits June 26, 2025 13:56

check=True, catch error instead

413b463

Merge branch 'master' into PYTHON-5253

75ddd14

sleepyStick requested a review from NoahStapp June 27, 2025 16:09

NoahStapp requested changes Jun 27, 2025

View reviewed changes

multiple patch files

1a3d87f

sleepyStick requested a review from a team as a code owner June 27, 2025 19:00

update contributing to explain patch files

4f3cb61

sleepyStick requested a review from NoahStapp June 27, 2025 19:27

sleepyStick added 2 commits July 11, 2025 09:35

Update CONTRIBUTING.md

1c901c7

remove unnecessary logic

380adbb

sleepyStick requested a review from blink1073 July 11, 2025 17:01

Merge branch 'master' into PYTHON-5253

7897afb

blink1073 previously approved these changes Jul 11, 2025

View reviewed changes

sleepyStick requested a review from ShaneHarvey July 11, 2025 20:23

NoahStapp requested changes Jul 14, 2025

View reviewed changes

Update CONTRIBUTING.md

92501a6

Co-authored-by: Noah Stapp <[email protected]>

sleepyStick dismissed blink1073’s stale review via 92501a6 July 14, 2025 16:19

sleepyStick and others added 9 commits July 14, 2025 09:19

Update CONTRIBUTING.md

49b0b40

Co-authored-by: Noah Stapp <[email protected]>

Update CONTRIBUTING.md

38f6ab9

Co-authored-by: Noah Stapp <[email protected]>

Update CONTRIBUTING.md

8938aa3

Co-authored-by: Noah Stapp <[email protected]>

Update CONTRIBUTING.md

7feb336

Co-authored-by: Noah Stapp <[email protected]>

Update CONTRIBUTING.md

0629bf9

Co-authored-by: Noah Stapp <[email protected]>

Update CONTRIBUTING.md

98c471c

Co-authored-by: Noah Stapp <[email protected]>

Update CONTRIBUTING.md

8108e98

Co-authored-by: Noah Stapp <[email protected]>

remove unnecessary exits

2c33abd

Merge branch 'PYTHON-5253' of https://github.com/sleepyStick/mongo-py…

02460d3

…thon-driver into PYTHON-5253

sleepyStick requested a review from NoahStapp July 14, 2025 16:31

NoahStapp requested changes Jul 14, 2025

View reviewed changes

sleepyStick added 2 commits July 14, 2025 10:32

move rm logging dir back to resync-spec

51f2a3f

add a comment for future

6bbf102

add exits back for spellcheck linting

fcc294c

sleepyStick requested a review from NoahStapp July 14, 2025 17:50

NoahStapp approved these changes Jul 14, 2025

View reviewed changes

blink1073 approved these changes Jul 14, 2025

View reviewed changes

		@@ -0,0 +1,1327 @@
		diff --git a/test/load_balancer/cursors.json b/test/load_balancer/cursors.json

PYTHON-5253 Automated Spec Test Sync #2409

Are you sure you want to change the base?

PYTHON-5253 Automated Spec Test Sync #2409

Conversation

sleepyStick commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sleepyStick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sleepyStick Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NoahStapp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blink1073 commented Jul 11, 2025

Uh oh!

sleepyStick commented Jul 11, 2025

Uh oh!

blink1073 commented Jul 11, 2025

Uh oh!

blink1073 left a comment

Choose a reason for hiding this comment

Uh oh!

NoahStapp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sleepyStick Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sleepyStick commented Jun 25, 2025 •

edited

Loading

sleepyStick Jun 26, 2025 •

edited

Loading

sleepyStick Jul 14, 2025 •

edited

Loading